Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[pull] master from restic:master #201

Open
wants to merge 1,160 commits into
base: master
Choose a base branch
from
Open

[pull] master from restic:master #201

wants to merge 1,160 commits into from

Conversation

pull[bot]
Copy link

@pull pull bot commented Feb 11, 2024

See Commits and Changes for more details.


Created by pull[bot]

Can you help keep this open source service alive? 💖 Please sponsor : )

@pull pull bot added the ⤵️ pull label Feb 11, 2024
greatroar and others added 29 commits October 5, 2024 10:56
This does not produce exactly the same messages, as it inserts newlines
instead of "; ". But given how long our error messages can be, that
might be a good thing.
…/x/sys-0.25.0

build(deps): bump golang.org/x/sys from 0.24.0 to 0.25.0
…g/automaxprocs-1.6.0

build(deps): bump go.uber.org/automaxprocs from 1.5.3 to 1.6.0
internal/restic: Use IDSet.Clone + use maps package
Bumps [golang.org/x/oauth2](https://github.com/golang/oauth2) from 0.22.0 to 0.23.0.
- [Commits](golang/oauth2@v0.22.0...v0.23.0)

---
updated-dependencies:
- dependency-name: golang.org/x/oauth2
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>
tag: fix swallowed error if repository cannot be opened
Changelogs should omit problem if its description duplicates the new behavior
generate: allow passing `-` for stdout output
…/minio/minio-go/v7-7.0.77

build(deps): bump github.com/minio/minio-go/v7 from 7.0.76 to 7.0.77
…/x/oauth2-0.23.0

build(deps): bump golang.org/x/oauth2 from 0.22.0 to 0.23.0
Bumps [google.golang.org/api](https://github.com/googleapis/google-api-go-client) from 0.195.0 to 0.199.0.
- [Release notes](https://github.com/googleapis/google-api-go-client/releases)
- [Changelog](https://github.com/googleapis/google-api-go-client/blob/main/CHANGES.md)
- [Commits](googleapis/google-api-go-client@v0.195.0...v0.199.0)

---
updated-dependencies:
- dependency-name: google.golang.org/api
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>
…ang.org/api-0.199.0

build(deps): bump google.golang.org/api from 0.195.0 to 0.199.0
Some error handling patches
dump: Simplify writeNode and use fewer goroutines
…ndows

fs: fix restoring timestamps on older Windows versions for long paths
restic cannot backup irregular files as those don't behave like normal
files. Thus skip them with an error.
backup: fix handling of files with type irregular
dump: add --compress flag to compress archives
fs: move NodeFromFileInfo into FS interface
Ignore AuthorizationFailure caused by using a container level SAS/SAT
token when calling GetProperties during the Create() call. This is because the
GetProperties call expects an Account Level token, and the container
level token simply lacks the appropriate permissions. Supressing the
Authorization Failure is OK, because if the token is actually invalid,
this is caught elsewhere when we try to actually use the token to do
work.
Add changelog entry in the 'unreleased' sub-folder for changes
introduced when fixing issue #4004.
MichaelEischer and others added 30 commits January 13, 2025 21:24
backup: allow excluding online-only cloud files
Stabilize `explicit-s3-anonymous-auth` and `safe-forget-keep-tags` feature flags
This fixes an issue where restic cannot find the tree when trying to find the
tree id of a snapshot.

---------

Co-authored-by: Albin Vass <[email protected]>
Co-authored-by: Michael Eischer <[email protected]>
Those methods now only allow modifying snapshots. Internal data types
used by the repository are now read-only. The repository-internal code
can bypass the restrictions by wrapping the repository in an
`internalRepository` type.

The restriction itself is implemented by using a new datatype
WriteableFileType in the SaveUnpacked and RemoveUnpacked methods. This
statically ensures that code cannot bypass the access restrictions.

The test changes are somewhat noisy as some of them modify repository
internals and therefore require some way to bypass the access
restrictions. This works by capturing an `internalRepository` or
`Backend` when creating the Repository using a test helper function.
…cked

repository: restrict SaveUnpacked and RemoveUnpacked
* tag: output the original ID and new snapshotID

tag: print changed snapshot information immediately

* print changed snapshot immediately after it has been saved
* add message type to the changedSnapshot
* add a summary type which will share the JSON output of the numer of changed snapshots
* updated verbosity of the changed snapshot in text mode to only work when verbosity > 2
* also use the terminal status printer for a standard handling for stdout messages
…tore

Allow excluding xattrs at restore time
The size comparison for `--max-unused` only accounted for unused but not
for duplicate data. For repositories with a large amount of duplicates
this can result in a situation where no data gets pruned even though
the amount of unused data is much higher than specified.
…plicates

prune: correctly account for duplicates in max-unused check
Removing the shadowing cases leads to weird workarounds but doesn't help
much with code clarity.
CI: bump golangci-lint and silence warnings about shadowed builtins
…5184)

The old sorting behaviour was to sort snapshots from oldest to newest.
The new sorting order is from newest to oldest. If one wants to revert to the
old behaviour, use the option --reverse.

---------

Co-authored-by: Michael Eischer <[email protected]>
Bumps [github.com/spf13/pflag](https://github.com/spf13/pflag) from 1.0.5 to 1.0.6.
- [Release notes](https://github.com/spf13/pflag/releases)
- [Commits](spf13/pflag@v1.0.5...v1.0.6)

---
updated-dependencies:
- dependency-name: github.com/spf13/pflag
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>
Bumps [github.com/Azure/azure-sdk-for-go/sdk/azidentity](https://github.com/Azure/azure-sdk-for-go) from 1.8.0 to 1.8.1.
- [Release notes](https://github.com/Azure/azure-sdk-for-go/releases)
- [Changelog](https://github.com/Azure/azure-sdk-for-go/blob/main/documentation/release.md)
- [Commits](Azure/azure-sdk-for-go@sdk/azcore/v1.8.0...sdk/azidentity/v1.8.1)

---
updated-dependencies:
- dependency-name: github.com/Azure/azure-sdk-for-go/sdk/azidentity
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>
…/Azure/azure-sdk-for-go/sdk/azidentity-1.8.1

build(deps): bump github.com/Azure/azure-sdk-for-go/sdk/azidentity from 1.8.0 to 1.8.1
Bumps [github.com/Azure/azure-sdk-for-go/sdk/storage/azblob](https://github.com/Azure/azure-sdk-for-go) from 1.5.0 to 1.6.0.
- [Release notes](https://github.com/Azure/azure-sdk-for-go/releases)
- [Changelog](https://github.com/Azure/azure-sdk-for-go/blob/main/documentation/release.md)
- [Commits](Azure/azure-sdk-for-go@sdk/azcore/v1.5.0...sdk/azcore/v1.6.0)

---
updated-dependencies:
- dependency-name: github.com/Azure/azure-sdk-for-go/sdk/storage/azblob
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>
…/spf13/pflag-1.0.6

build(deps): bump github.com/spf13/pflag from 1.0.5 to 1.0.6
ui/termstatus: Remove unused bytes.Buffer
…/Azure/azure-sdk-for-go/sdk/storage/azblob-1.6.0

build(deps): bump github.com/Azure/azure-sdk-for-go/sdk/storage/azblob from 1.5.0 to 1.6.0
)

* feat(backends/s3): add warmup support before repacks and restores

This commit introduces basic support for transitioning pack files stored
in cold storage to hot storage on S3 and S3-compatible providers.

To prevent unexpected behavior for existing users, the feature is gated
behind new flags:

- `s3.enable-restore`: opt-in flag (defaults to false)
- `s3.restore-days`: number of days for the restored objects to remain
  in hot storage (defaults to `7`)
- `s3.restore-timeout`: maximum time to wait for a single restoration
  (default to `1 day`)
- `s3.restore-tier`: retrieval tier at which the restore will be
  processed. (default to `Standard`)

As restoration times can be lengthy, this implementation preemptively
restores selected packs to prevent incessant restore-delays during
downloads. This is slightly sub-optimal as we could process packs
out-of-order (as soon as they're transitioned), but this would really
add too much complexity for a marginal gain in speed.

To maintain simplicity and prevent resources exhautions with lots of
packs, no new concurrency mechanisms or goroutines were added. This just
hooks gracefully into the existing routines.

**Limitations:**

- Tests against the backend were not written due to the lack of cold
  storage class support in MinIO. Testing was done manually on
  Scaleway's S3-compatible object storage. If necessary, we could
  explore testing with LocalStack or mocks, though this requires further
  discussion.
- Currently, this feature only warms up before restores and repacks
  (prune/copy), as those are the two main use-cases I came across.
  Support for other commands may be added in future iterations, as long
  as affected packs can be calculated in advance.
- The feature is gated behind a new alpha `s3-restore` feature flag to
  make it explicit that the feature is still wet behind the ears.
- There is no explicit user notification for ongoing pack restorations.
  While I think it is not necessary because of the opt-in flag, showing
  some notice may improve usability (but would probably require major
  refactoring in the progress bar which I didn't want to start). Another
  possibility would be to add a flag to send restores requests and fail
  early.

See #3202

* ui: warn user when files are warming up from cold storage

* refactor: remove the PacksWarmer struct

It's easier to handle multiple handles in the backend directly, and it
may open the door to reducing the number of requests made to the backend
in the future.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.